Dynamic texture and scene classification by transferring deep image features

摘要

Dynamic texture and scene classification are two fundamental problems inunderstanding natural video content. Extracting robust and effective featuresis a crucial step towards solving these problems. However the existingapproaches suffer from the sensitivity to either varying illumination, orviewpoint changing, or even camera motion, and/or the lack of spatialinformation. Inspired by the success of deep structures in imageclassification, we attempt to leverage a deep structure to extract feature fordynamic texture and scene classification. To tackle with the challenges intraining a deep structure, we propose to transfer some prior knowledge fromimage domain to video domain. To be specific, we propose to apply awell-trained Convolutional Neural Network (ConvNet) as a mid-level featureextractor to extract features from each frame, and then form a representationof a video by concatenating the first and the second order statistics over themid-level features. We term this two-level feature extraction scheme as aTransferred ConvNet Feature (TCoF). Moreover we explore two differentimplementations of the TCoF scheme, i.e., the \textit{spatial} TCoF and the\textit{temporal} TCoF, in which the mean-removed frames and the differencebetween two adjacent frames are used as the inputs of the ConvNet,respectively. We evaluate systematically the proposed spatial TCoF and thetemporal TCoF schemes on three benchmark data sets, including DynTex, YUPENN,and Maryland, and demonstrate that the proposed approach yields superiorperformance.